{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3c76c3c",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "#  30\\.  密度聚类标记异常共享单车  # \n",
    "\n",
    "##  30.1.  介绍  # \n",
    "\n",
    "本次挑战将考察密度聚类的应用。我们将尝试使用密度聚类可视化共享单车位置分布，同时使用不同的参数来标记出异常共享单车的位置。 \n",
    "\n",
    "##  30.2.  知识点  # \n",
    "\n",
    "  * DBSCAN 参数确定 \n",
    "\n",
    "  * HDBSCAN 聚类 \n",
    "\n",
    "如今，共享单车已经遍布大街小巷，的确方便了市民的短距离出行。不过，如果你是一家共享单车公司的运营，是否会考虑这样一个问题，那就是公司投放到城市中的共享单车都去哪里了呢？ \n",
    "\n",
    "当然，这个问题并不是为了满足你的好奇心，而是通过追踪共享单车的分布状况及时调整运营策略。比如，有一些位置的单车密度过高，那么就应该考虑将其移动到一些密度低但有需求的区域。 \n",
    "\n",
    "所以，今天的挑战中，将会使用到密度聚类方法来追踪共享单车的分布情况。 \n",
    "\n",
    "我们获取到北京市某一区域的共享单车 GPS 散点数据集，该数据集名称为 ` challenge-9-bike.csv  ` 。首先，加载并预览该数据集。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "112c1f27",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "wget -nc https://cdn.aibydoing.com/aibydoing/files/challenge-9-bike.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31521497",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.read_csv(\"challenge-9-bike.csv\")\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9710a0cb",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "|  lat  |  lon   \n",
    "---|---|---  \n",
    "count  |  3000.000000  |  3000.000000   \n",
    "mean  |  39.908308  |  116.474630   \n",
    "std  |  0.007702  |  0.018098   \n",
    "min  |  39.893939  |  116.434264   \n",
    "25%  |  39.902769  |  116.461276   \n",
    "50%  |  39.907888  |  116.477683   \n",
    "75%  |  39.914482  |  116.490274   \n",
    "max  |  39.923023  |  116.501467   \n",
    "  \n",
    "其中， ` lat  ` 是 latitude 的缩写，表示纬度， ` lon  ` 是 longitude 的缩写，表示经度。于是，我们就可以通过 Matplotlib 绘制出该区域共享单车的分布情况。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0155e2a",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure(figsize=(15, 8))\n",
    "plt.scatter(df[\"lat\"], df[\"lon\"], alpha=0.6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "821d940c",
   "metadata": {},
   "outputs": [],
   "source": [
    "<matplotlib.collections.PathCollection at 0x121cb4ac0>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3ebc4dd",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "[ ![../_images/7f7edd47cecfbc4f4aa3636e76df50a9dc6d9ea9f3e28abcd32386ef5a0848e2.png](../_images/7f7edd47cecfbc4f4aa3636e76df50a9dc6d9ea9f3e28abcd32386ef5a0848e2.png) ](../_images/7f7edd47cecfbc4f4aa3636e76df50a9dc6d9ea9f3e28abcd32386ef5a0848e2.png)\n",
    "\n",
    "接下来，我们尝试使用 DBSCAN 密度聚类算法对共享单车进行聚类，看一看共享单车高密度区域的分布情况。(可能会失效，对挑战无影响) \n",
    "\n",
    "根据前一节实验可知，DBSCAN 算法的两个关键参数是 ` eps  ` 和密度阈值 ` MinPts  ` 。那么这两个值设定为多少比较合适呢？ \n",
    "\n",
    "Exercise 30.1 \n",
    "\n",
    "挑战：使用 DBSCAN 算法完成共享单车 GPS 散点数据密度聚类，需要确定 ` eps  ` 和 ` min_samples  ` 参数。 \n",
    "\n",
    "规定：假设半径 100 米范围内有 10 辆车为高密度区域。 \n",
    "\n",
    "提示：挑战以纬度变化为参考，初略估算纬度变化 1 度，对应该区域 100km 的地面距离。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85e29876",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import DBSCAN\n",
    "\n",
    "## 代码开始 ### (≈ 2 行代码)\n",
    "\n",
    "## 代码结束 ###\n",
    "dbscan_c  # 输出聚类标签"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63802a51",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "参考答案  Exercise 30.1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c13af4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import DBSCAN\n",
    "\n",
    "### 代码开始 ### (≈ 2 行代码)\n",
    "dbscan_m = DBSCAN(eps=0.001, min_samples=10)\n",
    "dbscan_c = dbscan_m.fit_predict(df)\n",
    "### 代码结束 ###\n",
    "dbscan_c  # 输出聚类标签"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f28b2dbf",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "运行测试 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88b3b1bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(dbscan_c)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf02c8e6",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "期望输出 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eedbd07c",
   "metadata": {},
   "outputs": [],
   "source": [
    "6.977333333333333"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5df2120a",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "Exercise 30.2 \n",
    "\n",
    "挑战：针对上面聚类后数据，按要求重新绘制散点图。 \n",
    "\n",
    "规定：未被聚类的异常点以 ` alpha=0.1  ` 蓝色数据点呈现，聚类数据按类别呈现且设置 ` cmap='viridis'  ` 。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4044fdce",
   "metadata": {},
   "outputs": [],
   "source": [
    "## 代码开始 ### (≈ 4~8 行代码)\n",
    "plt.figure(figsize=(15, 8))\n",
    "## 代码结束 ###"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46923072",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "参考答案  Exercise 30.2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86c9ab1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "### 代码开始 ### (≈ 4~8 行代码)\n",
    "plt.figure(figsize=(15, 8))\n",
    "df_c = pd.concat([df, pd.DataFrame(dbscan_c, columns=['clusters'])], axis=1)\n",
    "\n",
    "df_n = df_c[df_c['clusters']!=-1]\n",
    "df_o = df_c[df_c['clusters']==-1]\n",
    "\n",
    "plt.figure(figsize=(15,8))\n",
    "plt.scatter(df_n['lat'], df_n['lon'], c=df_n['clusters'], cmap='viridis')\n",
    "plt.scatter(df_o['lat'], df_o['lon'], alpha=.1, c='b')\n",
    "### 代码结束 ###"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b142c727",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "期望输出 \n",
    "\n",
    "[ ![https://cdn.aibydoing.com/aibydoing/images/document-uid214893labid6102timestamp1531806489365.png](https://cdn.aibydoing.com/aibydoing/images/document-uid214893labid6102timestamp1531806489365.png) ](https://cdn.aibydoing.com/aibydoing/images/document-uid214893labid6102timestamp1531806489365.png)\n",
    "\n",
    "从上图可以看出，不同区域的单车密度分布情况。 \n",
    "\n",
    "HDSCAN 算法很多时候不仅仅是完成聚类，由于其本身的特性，很多时候还用其识别异常点。在本次实验中，我们同样可以通过调节参数来识别位置异常的共享单车。 \n",
    "\n",
    "Exercise 30.3 \n",
    "\n",
    "挑战：针对聚类后数据，将异常点（不符合半径 100 米内有 2 辆共享单车）绘制到散点图。 \n",
    "\n",
    "规定：未被聚类的边界点以红色数据点呈现，聚类数据按类别呈现且设置 ` alpha=0.1  ` ， ` cmap='viridis'  ` 。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32e11f8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "## 代码开始 ### (≈ 6~10 行代码)\n",
    "plt.figure(figsize=(15, 8))\n",
    "## 代码结束 ###"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ae25cf3",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "source": [
    "参考答案  Exercise 30.3 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c12ff603",
   "metadata": {},
   "outputs": [],
   "source": [
    "### 代码开始 ### (≈ 6~10 行代码)\n",
    "plt.figure(figsize=(15, 8))\n",
    "plt.figure(figsize=(15,8))\n",
    "\n",
    "dbscan_m = DBSCAN(eps=0.001, min_samples=2)\n",
    "dbscan_c = dbscan_m.fit_predict(df)\n",
    "dbscan_c\n",
    "\n",
    "df_c = pd.concat([df, pd.DataFrame(dbscan_c, columns=['clusters'])], axis=1)\n",
    "\n",
    "df_n = df_c[df_c['clusters']!=-1]\n",
    "df_o = df_c[df_c['clusters']==-1]\n",
    "\n",
    "plt.scatter(df_n['lat'], df_n['lon'], c=df_n['clusters'], cmap='viridis', alpha=.1)\n",
    "plt.scatter(df_o['lat'], df_o['lon'], c='r')\n",
    "### 代码结束 ###"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8977e7",
   "metadata": {},
   "source": [
    "期望输出 \n",
    "\n",
    "[ ![https://cdn.aibydoing.com/aibydoing/images/document-uid214893labid6102timestamp1531806489629.png](https://cdn.aibydoing.com/aibydoing/images/document-uid214893labid6102timestamp1531806489629.png) ](https://cdn.aibydoing.com/aibydoing/images/document-uid214893labid6102timestamp1531806489629.png)\n",
    "\n",
    "本次挑战主要是了解了如何快速确定 DBSCAN 初始参数以及使用该算法标记离群点的方法。如果你有兴趣，还可以自行尝试使用 HDBSCAN 聚类，并对比二者的聚类效果。当然，在这之前你需要先使用实验中的方法安装 hdbscan 模块。 "
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "main_language": "python",
   "notebook_metadata_filter": "-all"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
